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Prefatory  Note 


This  paper  on  leadership  skills  was  presented  at  the 
CONARC  Soft  Skills  Training  Conference,  sponsored  by  the 
U.S.  Continental  Army  Command  at  the  Air  Defense 
School,  Fort  Bliss,  Texas,  on  12-13  December  1972.  The 
purpose  of  the  conference  was  to  foster  an  extensive  inter¬ 
change  among  CONARC  representatives  and  invited  partic¬ 
ipants  regarding  approaches  to  the  systems  engineering  of 
soft  skills  training. 

Since  joining  the  Human  Resources  Research  Organi¬ 
zation,  Dr.  Jacobs  has  specialized  in  leadership  research  and 
leadership  training  development.  As  Director  of  HumRRO 
Division  No.  4  at  Fort  Benning,  Georgia,  he  is  presently 
responsible  for  direction  of  its  research  program  in  motiva¬ 
tion,  leadership,  and  both  group  and  individual  effectiveness. 


The  Evaluation  of  Leadership  Skills 

T.O.  Jacobs 


The  primary  purpose  of  this  paper  is  to  discuss  some  of  the  problems  involved  in 
the  evaluation  of  leadership  skills.  However,  it  will  not  be  possible  to  discuss  evaluation 
of  leadership  skills  without  first  considering  the  current  state  of  the  art  in  their 
identification  and  development. 


SOFT  SKILLS  SYSTEMS  ENGINEERING? 

This  frankly  provocative  title  was  selected  to  draw  attention  to  the  probable  source 
of  current  problems  in  quality  control  of  soft  skills  training.  A  basic-  problem,  at  least 
insofar  as  leadership  is  concerned,  is  that  the  systems  engineering  of  leadership  training  is 
only  beginning  to  emerge  from  the  dark  ages.  As-  this  paper  will  show,  there  is  a 
substantial  lack  of  agreement  on  at  least  the  following  points: 

(1)  What  leadership  is. 

(2)  What  affects  leadership  in  organizations. 

(3)  The  training  experience  that  will  produce  improved  leadership. 

(4)  Satisfactory  measures  of  leadership  ability. 

Difficulties  have  been  encountered  in  the  systems  engineering  of  training. 
Crawford  (1)  has  described  seven  steps  in  the  development  of  training,  as  shown  in 
Figure  1. 

A  case  can  be  made  that  the  major  problem  in  soft  skills  systems  engineering  lies  in 
the  second  step,  analysis  of  the  particular  job.  In  his  discussion  of  this  step,  Crawford 
says,  “We  are  interested  in  the  inputs  to  the  job— the  kind  of  stimuli  and  requirements 
placed  on  the  individual— and  the  outputs  from  the  job— what  the  man  does  to  bring 
about  system  output.”  This,  of  course,  is  a  reasonable  assertion.  The  problem  arises  when 
one  attempts  to  implement  this  step.  While  Crawford’s  paper  is  not  intended  to  be  a 
“how-to-do-it,”  it  is  instructive  to  note  that  a  rough  count  shows  that  only  91  words 
were  allocated  to  describe  this  step. 

It  appears  to  the  present  writer  that  systems  engineering  of  training  implicitly 
assumes  that  a  “true  expert”  is  to  be  found  somewhere,  and  that,  once  located,  he  will 
be  able  to  specify  his  inputs  and  outputs.  There  may  also  be  an  assumption  that  there  is 
a  single  correct  way  to  do  the  soft  skills  job.  In  the  remainder  of  this  paper,  all  of  these 
possible  assumptions  will  be  questioned. 


WHO  IS  THE  TRUE  EXPERT? 

Without  desiring  to  be  critical  of  an  interesting  and  innovative  beginning,  it  is 
instructive  to  search  momentarily  for  a  true  expert.  Whitmore  (2),  in  discussing  his 
concept  of  the  job  model,  notes  that  “The  identification  and  definition  of  job  functions 
from  system  characteristics  and  the  formulation  of  the  behavioral  processes  underlying 
each  function  requires  behavioral  science  expertise.”  This  suggests  the  possibility  that  the 
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Steps  in  the  Development  of  Training 


Figure  1 


behavioral  scientist  may  be  the  “true  expert,”  at  least  in  this  complex  soft  skills  area. 
The  validity  of  this  assumption  may  be  illustrated  by  a  quick  comparison  of  T-group 
protagonists  and  antagonists. 

T-group  (sensitivity)  training  is  a  fairly  widely  used  form  of  training  to  which 
executives  and  leaders  frequently  are  exposed  in  the  belief  that  it  will  “develop”  them. 
Bradford  (3),  as  one  protagonist,  describes  how  sensitivity  training  works.  He  concludes 
that,  while"  not  all  individuals  profit  equally  from  such  training,  most  benefit  to  some 
extent  and  “  . . .  the  majority  of  participants  have  learned  and  grown.” 

Adair  (4),  on  the  other  hand,  criticizes  sensitivity  training  on  four  accounts: 

(1)  Such  groups  have  no  formal  tasks,  which  means  that  they  do  not  have  the 
task  and  team  maintenance  responsibilities  of  real  groups  in  formal  organizations.  Thus, 
the  experience  is  dissimilar  to  the  on-the-job  conditions  of  performance,  which  should 
decrease  transfer  of  training. 

(2)  Several  assumptions  underlying  this  approach  probably  are  inaccurate.  An 
example  is  the  assumption  that  leadership  functions  should  be  distributed  equally  among 
the  members  of  the  group.  Adair  suggests  the  possibility  that  this  may  well  not  be  true 
when  there  are  real  tasks  which  must  be  achieved  under  diffi.u’t  situations. 

(3)  This  approach  as  a  universal  method  of  leadership  training  is  culture-bound. 
(It  should  be  noted  that  this  criticism  is  not  valid  if  transer  of  training  is  planned  only  to 
the  culture  to  which  it  is  “bound.”) 

(4)  There  is  a  tendency  for  group  psychotherapy  to  invade  the  training  scene. 
With  this  invasion  may  come  “preconceptions”  which  may  not  be  appropriate  for  leaders 
of  “normal”  people. 
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The  above  attempt  to  pit  two  schools  of  thought  against  one  another  is,,  admittedly, 
somewhat  unfair.  Clearly,  there  would  be  points  of  agreement.  However,  the  major  point 
is  that  there  clearly  are  differences  in  objectives,  implied  or  explicit.  That  is,  protagonists 
would  identify  one  set  of  objectives  for  leader  training  and  antagonists  another  set.  The 
differences  in  objectives  produce  differences  in  the  training  approach  taken.  Further,  the 
criteria  that  would  be  used  to  evaluate  the  effectiveness  of  the  training  would 
be  different. 

Clearly,  the  “true  expert”  may  or  may  not  be  the  behavioral  scientist.  Further,  to 
the  extent  that  he  has  theoretical  biases,  his  degree  of  “expertness”  may  be  com¬ 
promised.  And  1  believe  that  no  one  would  assert  that  behavioral  sciences  are  lacking  in 
theoretical  biases. 

There  is  yet  another  reasoii  why  the  behavioral  scientist  may  not  be  the  “true 
expert.”  Most  behavioral  scientists,  as  McGregor  (5)  points  out,  have  never  been  required 
to  perform  in  the  role  of  a  line  executive.  McGregor,  without  such  experience,  had 
believed  that  a  leader  could  operate  essentially  as  an  organizational  advisor,  and  that  good 
human  relations  would  be  quite  effective  in  relieving  discord  and  disagreement.  Experi¬ 
ence  demonstrated  to  him  that  a  leader  cannot  avoid  the  exercise  of  authority.  The 
leader  cannot  delegate  the  responsibility  which  he  alone  must  assume  for  the  workings  of 
his  organization. 

This  point  has  been  made  with  trepidation,  and  with  the  anticipation  of  challenge.  It 
is  in  fact  correct  that  experience  in  the  role  is  not  necessary  in  order  to  be  able  to 
describe  the  role  through  rigorous  observational  techniques.  So  some  behavioral  scientists 
may  be  able  to  rise  beyond  inexperience  to  become  “true  experts.”  However,  it  is  very 
difficult  to  observe  intangibles.  Leadership  and  executive  skills  are  intangible.  The 
necessary  recourse,  therefore,  is  to  study  experimentally  the  impact  of  variation  in  role 
behaviors  on  organizational  outcomes. 

It  should  be  recognized  that  this  is  an  extremely  difficult  proposition.  Several 
questions  immediately  are  raised,  which  bear  not  only  on  systems  engineering  but  also 
on  evaluation: 

(1)  What  should  be  used  as  the  criterion?  Should  it  be  evaluations  by 
superiors?  Should  it  be  evaluations  by  subordinates?  Should  it  be  some  rating  or  objective 
measure  of  organizational  performance? 

(2)  What  organization  can  be  made  available  for  experimental  study?  Real 
organizations  exist  in  a  world  that  usually  is  competitive.  Only  rarely  are  they  available 
for  experimental  study.  (And  they  also  are  hard  to  simulate.) 

(3)  How  much  time  can  be  devoted  to  experimental  study?  The  time  required 
varies,  depending  on  the  object  of  the  study.  In  a  cross-sectional  study,  relatively  little 
time  is  required.  In  a  longitudinal  study,  months  or  perhaps  even  years  may  be  required. 


WHAT  IS  THE  NATURE  OF  THE  REAL  WORLD? 

Bruner  (6)  discusses  two  different  kinds  of  concepts.  The  first,  conjunctive  concepts, 
consists  of  those  in  which  the  logic  is  essentially  transitive.  That  is,  if  in  Middletown  only 
residents  can  vote,  it  is  possible  to  infer  by  knowing  that  an  individual  has  voted  that  he 
is  a  resident.  Man-machine  systems  basically  are  conjunctive  systems.  The  logic  of 
man-machine  interaction  is  a  transitive  logic. 

The  second  type  of  concept  is  disjunctive,  in  which  this  kind  of  transitivity  does  not 
occur.  For  example,  if  in  Middletown  either  residents,  taxpayers,  or  persons  who  work 
within  the  city  may  vote,  then  it  is  not  possible  to  infer  the  individual’s  category  from 
knowing  that  he  votes.  It  is  very  likely  that  most  man-ascendant  systems  are 
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characterized  by  disjunctive  logic,  where  several  different  solutions  may  work  in  any 
given  problem  situation.  (This  is  recognized  in  the  idiom  by  the  statement  that  any 
solution,  well  implemented,  is  better  than  no  solution.) 

Of  course,  it  is  possible  that  organizational  leadership  is  not  actually  disjunctive.  It 
may  only  appear  so  because  of  the  many  variables  that  may  affect  the  impact  of  any 
given  leadership  behavior  or  approach.  Further,  there  may  be  moderator  variables  that 
actually  change  the  relationship  between  a  given  leadership  behavior  and  the  expected 
outcome.  (Fiedler’s  contingency  model  illustrates  the  use  of  moderator  variables  in 
leadership  theory  buOding.)  Several  studies  illustrate  these  variables. 

Organizational  Climate.  In  a  relatively  well-known  study,  Fleishman  (7)  reported  the 
effects  of  leadership  training  for  foremen  after  the  foremen  had  returned  to  their 
industrial  situation.  The  leadership  training  itself  consisted  of  a  focus  on  initiating 
structure  and  showing  consideration  dimensions.  Fleishman  reported  that  the  effects  of 
the  training  appeared  rather  small  when  a  later  follow-up  was  made.  The  kind  of 
supervisor  (leadership  climate)  under  which  the  foreman  worked  was  more  related  to  his 
own  behavior  than  either  training  or  lack  of  it.  Foremen  working  under  a  considerate 
supervisor  tended  to  express  more  considerate  attitudes  toward  their  workers,  and 
received,  in  return,  higher  consideration  scores  from  their  workers. 

Organizational  Structure.  Forehand  and  Gilmer  (8)  further  analyzed  the  effect  of 
organizational  climate  on  leader  behavior.  They  noted  that  climate  may  be  determined 
not  only  by  the  leadership  pattern  of  one’s  own  supervisors,  but  also  by  size,  organiza¬ 
tional  structure,  systems  complexity,  and  goals.  Presumably,  any  of  these  factors  could 
negate  or  enhance  the  impact  of  leadership  training,  or  modify  the  relationship  between 
given  leadership  behavior  and  system  outcomes. 

Nonlinear  Relationships.  A  report  of  a  seminar  held  by  the  Foundation  for  Research 
on  Human  Behavior  (9)  describes  several  of  these  non-linear  relationships: 

Production  Pressure.  When  production  pressure  was  low,  an  increase  in  pressure 
increased  both  satisfaction  with  supervisor  and  proi’  "•tion,  to  a  point,  beyond  which 
satisfaction  dropped  sharply. 

Group  Cohesiveness.  Group  cohesion  is  positively  correlated  with  productivity 
when  the  company  is  perceived  as  supportive,  but  negatively  correlated  with  productivity 
when  the  company  is  perceived  as  threatening.  (This  illustrates  a  moderator  variable.) 

Closeness  of  Supervision.  Production  can  be  high  either  way,  but  morale  suffers 
with  excessively  close  supervision,  and  a  large  investment  of  supervisor  energy  must  be 
expended  also. 

Subordinate  Expectations.  Jasinski  (10)  reported  an  interesting  study  from  the 
automobile  industry,  in  which  workers  and  foremen  alike  had  proper  and  idealized 
descriptions  of  what  the  supervisor’s  role  ought  to  be.  However,  in  actual  practice  the 
foreman’s  behavior  was  very  different  from  the  verbal  description  of  the  ideal  role. 
Further,  Jasinski  found  that  workers  really  expected  a  different  type  of  behavior  from 
the  foreman  than  they  said  they  wanted,  and  the  work  environment  precluded  the 
lengthy  interaction  between  workers  and  foremen  that  would  have  been  required  by 
“desired”  behavior.  Finally,  the  supervisors  of  the  foremen  gave  little  weight  to  foreman 
relations  with  men  in  their  evaluations  of  the  foremen. 

When  leadership  is  taught  and  evaluated  in  a  service  school,  the  service  school 
essentially  is  serving  a  customer.  The  customer  is  the  commander  in  the  field  who  uses 
the  product  generated  by  the  school.  It  clearly  is  important  to  develop  and  evaluate  the 
right  thing.  One  further  example  of  the  cataclysmic  results  that  may  occur  when  the 
wrong  thing  is  developed  is  found  in  a  report  by  Sikes  (H)  of  a  course  on  human 
relations.  This  course  was  given  to  all  managers  and  supervisors  in  a  medium-sized 
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contracting  firm.  The  evaluation  tool  was  a  questionnaire  to  supervisors.  One  question 
was,  “Was  the  course  as  a  whole  a  success  or  a  failure”?  The  summary  of  answers  was: 

Success  0 
Failure  83 

No  Opinion  14 

These  startling  findings  occurred  because  participants  felt  the  course  had  not  made 
any  lasting  improvement  in  the  attitudes  of  senior  management.  However,  the  cataclysmic 
further  result  was  a  crisis  in  the  conflict  between  expectations  held  by  supervisors  and  by 
senior  management  concerning  the  role  of  management.  This  conflict  led  to  resignations 
of  19  of  the  97  supervisors  who  had  attended  the  course.  (In  the  two  years  preceding  the 
course  only  two  had  left.)  Further,  25  more  were  known  to  have  applied  for  other  jobs. 


HOW  TO  MEASURE  THE  UNMEASURABLE? 

At  the  outset,  it  was  noted  that  ultimate  criteria  are  rarely  available  for  measuring 
the  impact  of  leadership  training  and  development.  There  are  probably  two  reasons.  The 
first  is  the  difficulty  of  obtaining  objective  criterion  measures  in  real  organizations; 
realistically,  this  does  present  problems.  The  second  difficulty  is  that  it  sometimes  is 
embarrassing  to  the  researcher  to  be  able  to  obtain’  such  criteria  when  they  are  available. 

Korman  (12)  distinguishes  two  kinds  of  validity,  concurrent  and  predictive.  Con¬ 
current  validity  is  derived  from  measurements  taken  at  the  same  time  that  measurements 
are  made  of  the  so-called  leader  manipulation.  Predictive  validity  is  derived  from  a 
longitudinal  study,  in  which  a  leadership  manipulation  is  made,  and  efforts  are  then  made 
to  determine  whether  individuals  in  the  experimental  group  behave  in  different  (hopefully 
better)  ways  on  the  job. 

The  embarrassing  aspect  of  such  criteria  is  that  they  frequently  lead  to  the  con¬ 
clusion  that  no  change  has  occurred,  especially  where  the  criteria  consist  of  objective 
measures  of  organizational  performance.  It  will  be  recalled  that  Fleishman’s  finding  of  no 
impact  came  from  a  longitudinal  study.  In  his  own  review  of  the  literature,  Korman 
found  quite  similar  results.  His  focus  was  on  Initiating  Structure  and  Showing  Consid¬ 
eration  as  leadership  behavior  variables.  Most  of  the  studies  reported  concurrent  validity, 
which  Korman  notes  does  not  give  good  evidence  of  causality.  Further,  the  objective 
criteria,  when  they  were  used,  yielded  low  correlations  with  both  Consideration  and 
Initiating  Structure.  This  is  a  finding  that  is,  of  course,  repeated  fairly  frequently  in 
other  studies. 

Korman’s  other  results  are  interesting  in  their  own  right.  Table  1  summarizes  his 
review.  It  shows  correlations  between  Initiating  Structure  and  Showing  Consideration 
variables,  and  a  variety  of  other  measures.  In  this  table  concurrent  and  predictive  validity 
data  have  been  mixed.  As  can  be  seen,  there  is  substantial  variability  in  the  findings, 
regardless  of  what  criterion  is  used.  The  only  consistent  findings  appear  to  be  correlations 
,  between  subordinate  ratings  of  leaders  and  Showing  Consideration  behavior.  Korman 
concludes  that  his  survey  of  the  literature  does  not  provide  any  reason  for  saying  that 
these  leader  behavior  variables  can  predict  work  group  performance,  or  under  what 
conditions  they  might  affect  work  group  performance,  a  somewhat  dismal  picture. 

These  findings  represent  only  a  marginal  sampling  from  the  available  literature. 
However,  they  do  nonetheless  illustrate  the  extreme  difficulty  of  determining  what  should 
be  taught  as  leadership,  how  it  should  be  taught,  and  how  the  effectiveness  of  the 
resulting  skill  should  be  evaluated. 
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Table  1 


Corral  it  ions  Batwaan  Various  Crrtaria  and 
Initiating  Structure  and  Showing  Conskfaration 


With 

Initiating  Structure 

Shoering  Consideration 

N 

Rang* 

N 

Range 

Objective  Measures 

5 

-.05  to  .23 

5 

-.11  to. 14 

Superior  Ratings 

8 

-.39  to  .13 

8 

-.06  to  .29 

Peer  Ratings 

8 

-.11  to  .14 

8 

-.17  to  .03 

Self  Ratings 

3 

-.41  to  .45 

3 

-.02  to -.46 

Subordinate  Ratings 

3 

.05  to  .22 

3 

.24  to  .51 

IF  NOT  ULTIMATE  CRITERIA,  THEN  WHAT? 

While  so-called  ultimate  criteria  may  be  available  within  organizational  settings,  they 
generally  are  not  available  to  the  trainer.  This  has  led  to  a  proliferation  of  intermediate 
criteria  which  have  been  used  for  assessment  purposes.  Some  of  the  vehicles  for 
leadership  assessment  have  been: 

Situational  Performance  Tests 

Situational  Paper-and-Pencil  Tests 

Personality  Tests 

Peer  Ratings 

Ratings  by  Seniors 

Leadership  Grades 

This  list  is  by  no  means  inclusive.  It  is  intended  primarily  to  illustrate  the  diversity  of 
approaches  in  current  use. 

Situational  Tests.  These  tests  generally  depend  for  predictive  validity  on  the 
similarity  between  the  performance  requirement  (including  group  leadership  and 
direction)  of  the  test  situation  and  that  of  the  ultimate  performance  situation.  However, 
situational  tests  have  been  plagued  by  many  common  psychometric  problems,  such  as: 

•  The  difficulty  of  establishing  parallel  situations,  so  that  different  leaders  will 
be  assessed  under  comparable  conditions  of  performance. 

•  The  difficulty  of  establishing  situations  which  in  fact  present  real  life  per¬ 
formance  requirements  at  other  than  a  face  valid  level. 

•  The  cost  in  terms  of  time  and  administrative  difficulty  in  running  good 
situational  tests. 

•  The  difficulty  of  training  assessors  so  that  they  respond  to  the  same  leader 
behaviors  in  the  same  way,  or  even  observe  the  same  leader 
behavior  variables. 

•  The  difficulty  of  providing  subordinates  who  respond  as  subordinates  would 
in  real  life.  (In  the  test  situation  they  are  not  subject  to  quite  the  same 
actual  leader  power  as  in  a  real  unit.) 

Situational  Paper-and-Pencil  Tests.  An  example  is  the  In-Basket  Test.  A  problem 
with  this  kind  of  test  is  that  one  of  the  most  critical  dimensions  of  leadership  skill 
probably  is  interpersonal  competence.  The  austere  context  of  an  In-Basket  Test  is  hardly 
appropriate  for  assessing  this  kind  of  variable.  Further,  in  the  military  setting,  there  may 
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be  some  degree  of  stress  associated  with  the  performance  requirement,  which  is  difficult 
to  simulate  in  an  in-basket  or  other  similar  test.  Finally,  it  is  impossible  to  assess  group 
maintenance  or  individual  support  leader  behavior  skills  in  a  “solo”  performance. 

Personality  Tests.  Typical  of  this  approach,  which  is  about  three  decades  old,  is  the 
self-assessment  test,  which  commonly  takes  the  form  of  measuring  the  individual’s 
proclivities  toward  authoritarian,  democratic,  or  laissez-faire  leadership.  The  fallacies  of 
this  approach  are  obvious. 

Peer  Ratings.  While  it  is  difficult  to  know  exactly  what  peer  ratings  measure,  they 
generally  have  had  better  predictive  validity  than  almost  any  other  single  approach.  This 
may  be  because  they  probably  measure  the  dimension  of  interpersonal  competence  and 
the  ability  to  work  within  a  coordinated  group  better  than  do  most  other  approaches. 
However,  even  so,  peer  ratings  rarely  account  lor  more  than  25%  of  the  total  variance  in 
subsequent  leadership  performance,  and  are  always  subject  to  the  vagaries  of  the  peers 
who  are  completing  them.  If  a  group  decides  to  do  so,  “cooperate  and  graduate”  can  be 
made  to  work  extremely  well.  Nonetheless,  peer  ratings  are  a  “method  of  choice,” 
especially  when  used  in  conjunction  with  other  measures. 

Ratings  by  Seniors.  It  is  very  difficult  to  assess  the  extent  to  which  ratings  by 
seniors  are  predictive  in  an  independent  sense.  That  is,  in  a  formal  school  setting,  peer 
ratings  and  tactical  officer  ratings  are  found  to  be  fairly  highly  correlated.  However, 
ratings  by  tactical  officers  are  generally  less  reliable  than  ratings  by  peers. 

Leadership  Grades.  Finally,  leadership  grades  have  been  developed  from  paper- 
and-pencil  knowledge  tests  following  leadership  instruction.  In  unpublished  research, 
HumRRO  Division  No.  4  has  found  them  on  occasion  even  to  be  negatively  correlated 
with  leadership  grades  obtained  through  a  combination  of  peer  ratings  and  tactical  officer 
ratings.  This  suggests  that  they  probably  would  not  predict  subsequent  performance,  in 
that  peer  ratings  generally  do. 


A  DISJUNCTIVE  APPROACH  FOR  A  DISJUNCTIVE  CONCEPT 

The  picture  to  this  point  has  been  negative,  and  unfairly  so.  It  was  presented  in  this 
fashion  to  illustrate  the  difficulties  in  both  soft  skills  systems  engineering,  and  the 
evaluation  of  soft  skills  such  as  leadership.  Soft  skills  systems  engineering  is  extraor¬ 
dinarily  difficult  because  of  the  absence  of  a  “true  expert,”  and  because  of  the  intangible 
nature  of  these  skills.  Because  the  evaluation  process  is  necessarily  dependent  on  identi¬ 
fication  of  what  is  to  be  evaluated,  the  development  of  soft  skill  evaluation  procedures 
has  necessarily  been  impeded  by  the  same  problems. 

However,  an  assertion  may  be  made  at  this  point.  While  it  is  extremely  unlikely  that 
any  massive  breakthrough  is  ever  going  to  occur  in  soft  skills  systems  engineering, 
approaches  such  as  the  behavioral  model  (Whitmore  (2))  are  conceptually  sound,  and  will 
eventually  produce  a  sound  picture  of  what  it  is  that  needs  to  be  trained.  It  will  then 
become  feasible  to  develop  evaluation  techniques.  The  requirement  is  for  patience  and 
continued  effort,  with  the  realization  that  this  is  simply  a  much  more  difficult  problem 
than  hard  skills  systems  engineering. 

As  an  example  of  possible  future  evaluation  approaches,  it  is  appropriate  to  examine 
the  “scientific  assessment  center.”  Management  assessment  centers  are  becoming  more 
frequent  in  industry,  probably  because,  as  Byham  (13)  asserts,  they  have  shown  them¬ 
selves  “. . .  to  be  a  more  reliable  indicator  of  future  success  than  any  other  tool 
yet  devised.” 

The  assessment  center  works,  probably,  because  it  is  so  comprehensive  in  what  it 
measures.  (Critics  might  argue  that  if  enough  of  an  individual’s  behavior  is  observed, 
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probably  it  would  be  possible  to  make  predictions  to.  any  future  activity.)  Byham 
identifies  25  “common  dimensions”  of  managerial  success: 


Listening  skill 
Flexibility 
Tenacity 
Risk  taking 
Initiative 
Independence 
Planning  and  organizing 
Management  control 
Use  of  delegation 
Problem  analysis 
Judgment 
Decisiveness 


Impact 
Energy 

Oral  communciation  skill 
Oral  presentation  skill 
Written  communication  skill 
Creativity 
Range  of  interest 
Stress  tolerance 
Motivation 
Work  standards 
Leadership 
Salesmanship 
Sensitivity 

Byham  also  identifies  eight  techniques  that  may  be  used  in  an  assessment  center  for 
evaluating  individuals  on  these  dimensions: 

Interview 

Management  Games 

In-Basket  and  Interview 

Leaderless  Group  Discussion  (Assigned) 

Leaderless  Group  Discussion  (Non-Assigned) 

Fact  Finding  and  Decision  Making 
Analysis  Presentation 
Interview  Simulation 

Some  may  be  better  than  others  for  assessing  one  or  more  of  the  preceding  dimensions. 

.  In  the  assessment  center,  the  optimum  measurement  approach  is  selected  for  each 
measurement  to  be  obtained. 

According  to  Byham,  psychologists  are  considerably  superior  to  untrained  line 
assessors  in  the  assessment  center  context.  However,  they  are  not  always  available,  so 
companies  tend  to  establish  pools  of  trained  assessors.  With  either,  the  predictive  validity 
of  the  assessment  center  is  quite  good.  Bray,  Grant,  and  Campbell  (14)  make  an 
essentially  similar  point. 

The  difficulty  with  assessment  centers  is,  of  course,  that  they  are  expensive  to 
operate,  and  have  a  low  throughput  of  assessed  potential  leaders.  Where  management 
turnover  is  low,  this  is  no  problem.  In  the  Army,  with  high  input  of  junior  leaders,  it 
might  create  difficulties. 

However,  in  the  context  of  the  present  paper,  the  assessment  center  concept  is 
probably  worth  examining  as  the  prototype  for  soft  skills  evaluation  procedures  develop¬ 
ment.  The  assessment  center  “covers  the  waterfront.”  Similarly,  it  is  probable  that  the 
evaluation  of  any  soft  skill  will  necessarily  cover  a  multitude  of  dimensions.  Single 
unitary  measures  probably  will  not  suffice.  (This  suggests  that  leadership  training  also  will 
need  to  cover  a  variety  of  dimensions.)  Finally,  it  is  likely  that  leadership  evaluation 
procedures  will  gradually  improve  with  experience  and  research,  just  as  leadership 
development  and  training  procedures  improve  through  the  same  means.  In  each  case, 
continued  effort  should  provide  increasingly  improved  payoffs. 

As  a  final  example  of  the  inextricable  intertwining  of  leadership  research  and 
leadership  development,  it  is  appropriate  to  examine  a  probable  new  direction  in  the 
assessment  of  organizational  leadership.  Students  of  small  group  theory  will  affirm  that 
substantial  progress  in  understanding  of  small  group  dynamics  came  only  after  researchers 
turned  to  the  study  of  processes  that  occur  in  small  groups.  Examination  of  the 
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assessment  dimensions  just  cited  shows  that  they  are  essentially  person-oriented.  Thus, 
they  are  likely  to  be  limited  in  their  predictive  power. 

However,  a  few  researchers  have  been  turning  to  a  study  of  organizational  processes 
in  order  to  understand  organizational  leadership  more  fully.  This  generally  requires  some 
sort  of  simulation  of  an  organization,  because  real  organizations  cannot  be  “replayed.*’ 
The  capacity  to  repeat  a  standardized  experience  is,  of  course,  necessary  in  order  to 
develop  a  basis  for  inferential  testing.  As  an  example  of  this  work,  Olmstead  (15)  has 
developed  a  simulation  of  a  battalion  engaged  in  a  counterinsurgency  operation.  The 
simulation  is  designed  to  permit  measurement  of  organizational  process  variables 
identified  by  Schein  (16)  as  the  Adaptive  Coping  Cycle.  These  variables  and  their 
correlations  with  independent  measures  of  the  adequacy  of  mission  accomplishment 
(Olmstead ’s  simulation)  are: 

Sensing  a  change  in  the  internal  or  external  environment  .92 


Communicating  the  information  to  a  decision  maker  .83 

Making  a  decision  with  the  information  .70 

Stabilizing  internal  changes  .11 

Communicating  change  decisions  .71 

Coping  actions  .72 

Obtaining  feedback  as  to  decision  effectiveness  .03 


It  is  beyond  the  scope  of  the  present  paper  to  present  Olmstead’s  findings  in  greater 
depth.  The  significant  point  is  that  organizational  leadership  possibly  should  be  viewed  as 
the  capacity  to  cause  certain  crucially  important  functions  to  be  performed  well.  This 
would  have  equally  significant  implications  for  assessment  of  leadership.  The  question 
would  be,  “Leadership  for  what?”  And  the  assessment  technique  would  be  examination 
of  the  leader’s  capacity  for  making  processes  work  well. 

As  a  contrast,  Olmstead  also  obtained  data  using  many  of  the  conventional  paper- 
and-pencil  measures,  with  miniscule  results.  In  this  simulation,  the  so-called  “personality” 
measures  used  did  not  tap  the  capacity  to  make  the  process  go  smoothly.  Also,  they  did 
not  tap  organizational  effectiveness.  The  implications  of  these  findings  for  assessment  of 
leadership  skills  are  profound. 


SUMMARY 

In  this  paper,  it  has  been  asserted  that  the  development  of  leadership  evaluation 
procedures  has  been  impeded  by  the  same  obstacles  that  have  slowed  the  development  of 
soft  skills  systems  engineering  techniques.  It  is  difficult  to  find  a  “true  expert,”  and  the 
incumbent  is  frequently  unable  to  verbalize  what  he  actually  does  because  what  he  does 
is  intangi’  ie.  This  means  that  knowledge  must  be  developed  through  the  arduous  method 
of  experimental  study  of  leadership  processes  by  researchers  and  other  observers.  How¬ 
ever,  the  general  unavailability  of  ultimate  criteria  of  leadership  effectiveness  makes  this  a 
slower  process  than  it  otherwise  might  be. 

The  dearth  of  ultimate  criteria  has  led  to  a  proliferation  of  intermediate  criteria.  The 
researcher’s  own  bias  has  often  influenced  the  intermediate  criteria  he  has  chosen  to 
study,  and  the  hypotheses  he  has  chosen  to  test.  Researcher  bias  has  also  influenced  the 
types  of  leadership  processes  he  has  chosen  to  study. 

Despite  the  handicaps  that  shackle  the  researcher,  among  which  is  the  fact  that  he 
rarely  has  been  in  the  position  of  a  line  executive  and  thus  does  not  know  how  it  feels, 
the  researcher  probably  is  indispensable  in  the  soft  skills  area,  as  Whitmore  (2)  has  noted. 
This  is  not  because  of  what  he  knows  substantively.  Indeed,  he  may  be  more  dangerous 
because  of  his  substantive  knowledge.  Rather,  the  researcher’s  value  stems  from  his 
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training  in  scientific  methodology  and  from  the  fact  that  he  is,  by  virtue  of  that  training, 
a  highly  qualified  observer-providing  that  his  own  theoretical  biases  do  not  blind  him. 

It  is  probable  that  most  soft  skill  areas  are  disjunctive  in  nature.  This  creates 
substantial  difficulties  for  anyone  attempting  to  do  systems  engineering  work.  However, 
substantial  progress  has  been  made  in  recent  years  in  the  development  of  knowledge 
about  leadership,  as  an  example  of  one  soft  skill.  The  assessment  center  concept  was 
discussed  briefly  as  a  possible  prototype  of  future  evaluation  approaches. 

Although  not  identified  specifically  in  the  paper,  there  are  needs  that  are 
worth  citing: 

(1)  Continued  research  to  define  the  substantive  nature  of  the  leadership 
phenomenon.  Such  research  will  continue  to  produce  progress  in  the  development  of 
leadership  training,  and  will  permit  parallel  progress  in  the  development  of  evaluation 
techniques. 

(2)  Continued  search  for  intermediate  criteria.  This  wi’l  require  the  availability  of 
organizations  for  study,  perhaps  through  simulation  techniques  such  as  those  Olmstead  (15) 
has  used.  Further,  there  is  a  need  lor  enlightened  decision  makers  who  will  tolerate  other 
than  face-valid  measures  for  leadership  assessment. 

(3)  Continued  work  to  learn  how  to  modify  an  individual  to  be  more  effective 
as  a  leader. 
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